Skip to content

qrep: derive pull columns from source instead of catalog schema - #4583

Merged
dtunikov merged 5 commits into
mainfrom
qrep-load-schema-without-watermark-setup
Jul 16, 2026
Merged

qrep: derive pull columns from source instead of catalog schema#4583
dtunikov merged 5 commits into
mainfrom
qrep-load-schema-without-watermark-setup

Conversation

@dtunikov

@dtunikov dtunikov commented Jul 14, 2026

Copy link
Copy Markdown
Contributor

Problem

QRep pulls with excluded columns or child-table range partitions fail with:

[qrep] failed to pull records: failed to load table schema: no rows in result set

The pull path (qrep_source.go) built the SELECT column list by loading the table schema from table_schema_mapping, keyed by (ParentMirrorName, DestinationTableIdentifier). When no row exists for that key, QueryRow().Scan() returns pgx.ErrNoRows ("no rows in result set") and the pull hard-fails.

This has been observed on a CDC mirror. The in-repo setup paths normally populate that row under a matching key, so the exact reason the row was absent for this mirror isn't pinned down (candidates: version skew / older mirror, or attaching to a pre-existing mirror). Regardless, the pull shouldn't depend on the catalog row being present when it can derive the columns directly from the source.

Fix

Derive the column list directly from the source via GetSelectedColumns (which reads pg_attribute) — the same primitive getTableSchemaForTable already uses to build the identical list, so column selection and ordering are unchanged.

This removes the table_schema_mapping dependency from the pull path entirely. catalogPool is no longer needed by corePullQRepRecords.

🤖 Generated with Claude Code

The QRep pull path loads the table schema from table_schema_mapping when
Exclude columns or child table ranges are set, but the row was only written
when SetupWatermarkTableOnDestination was true. Standalone QRep mirrors to
non-PG destinations with excluded columns hit "no rows in result set".

Move setupTableSchema out from behind the SetupWatermarkTableOnDestination
guard so the schema is always available to the pull path.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@dtunikov
dtunikov requested a review from a team as a code owner July 14, 2026 14:06
Only standalone mirrors (ParentMirrorName == FlowJobName) lack a CDC
setup_flow to populate table_schema_mapping; CDC initial-load children
already have it under ParentMirrorName. Guard the unconditional
setupTableSchema so CDC children don't run a redundant GetTableSchema.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@claude

claude Bot commented Jul 14, 2026

Copy link
Copy Markdown

Code review

No issues found. Checked for bugs and CLAUDE.md compliance.

The QRep pull path built the SELECT column list (for Exclude / child table
ranges) by loading the table schema from table_schema_mapping keyed by
(ParentMirrorName, DestinationTableIdentifier). That row can be absent for
CDC snapshots after resync/table-addition (the destination is renamed with a
_resync suffix), producing "failed to load table schema: no rows in result
set".

Use GetSelectedColumns to read the column list directly from the source's
pg_attribute, matching how getTableSchemaForTable already builds it. This
removes the catalog dependency and fixes standalone, CDC, resync and
table-addition paths alike. catalogPool is no longer needed by
corePullQRepRecords.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
@dtunikov dtunikov changed the title qrep: always populate table_schema_mapping for standalone mirrors qrep: derive pull columns from source instead of catalog schema Jul 14, 2026
@dtunikov
dtunikov requested a review from Copilot July 14, 2026 14:33

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Removes QRep pull-time dependence on table_schema_mapping by deriving the SELECT column list directly from the source table (via pg_attribute), preventing pulls from hard-failing when the catalog schema row is missing.

Changes:

  • Stop loading table schema from the catalog for building the SELECT column list during QRep pulls.
  • Derive selected columns from the source table using GetSelectedColumns when excludes or child-table range partitions are involved.
  • Drop catalogPool usage from corePullQRepRecords (callers keep the parameter but ignore it).

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread flow/connectors/postgres/qrep_source.go
Comment thread flow/connectors/postgres/qrep_source.go
AND c.relname = $2
AND a.attnum > 0
AND NOT a.attisdropped ` + excludedColumnsSQL
AND NOT a.attisdropped` + excludedColumnsSQL + ` ORDER BY a.attnum`

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

optional, suggested by copilot to keep columns order consistent across calls

@ilidemi ilidemi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would prefer to first understand what led to the error before making a change just in case. What if there's a consistency issue

@dtunikov

Copy link
Copy Markdown
Contributor Author

@ilidemi

Would prefer to first understand what led to the error before making a change just in case. What if there's a consistency issue.

I explained it in internal discussion, posting here as well:
this is what has happened most probably:

  • Resync cancels the active snapshot/QRep children.
  • DropFlowWorkflow deletes all table_schema_mapping rows, including during resync.
  • An already-running QRep pull gets far enough to perform this new schema lookup before its cancellation is observed.
  • It sees the deleted row and emits this exact error.

@dtunikov
dtunikov requested a review from ilidemi July 15, 2026 09:34

@ilidemi ilidemi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah right, I missed that thread. Conceptually having a single source of truth is nicer, and waiting on full Temporal cancellation before starting resync would be a cleaner fix (if possible), but this is also bringing us closer to not needing an overarching snapshot tx and still supporting schema changes so ok enough.

@dtunikov
dtunikov merged commit b8357f1 into main Jul 16, 2026
22 checks passed
@dtunikov
dtunikov deleted the qrep-load-schema-without-watermark-setup branch July 16, 2026 16:37
ilidemi added a commit that referenced this pull request Aug 5, 2026
### Error (sanitized)

Postgres snapshot partitions fail during `qrep` pull with a claim that
the source table has no columns:

```
[qrep] failed to pull records: table "public"."ferly_inoculum" doesn't have queriable columns
```

The line emits `errorClass=OTHER` / `errorCode=UNKNOWN` with source
`other`. It has appeared on several pipes across multiple regions over
the past few weeks, always in a short burst that clears within minutes.

### Investigation

This a bug in PeerDB. `GetSelectedColumns` iterates `pg_attribute` rows
but never checks `rows.Err()`, so when the source connection dies after
the query is sent, `Next` stops immediately and the function returns an
empty column list with a nil error. `corePullQRepRecords` reads that
empty list as "this table has no columns" and reports it, hiding the
connection failure that actually happened.

- [pgx
`Conn.Query`](https://pkg.go.dev/github.com/jackc/pgx/v5#Conn.Query) —
documents that `Query` returns before reading any rows, so a failure
after the query is sent surfaces only through `Rows.Err()`.
- [PeerDB #4583](#4583) — added
this pull path and its empty-column check; the message cannot predate
it, and the first occurrences follow it.
- [Postgres error
codes](https://www.postgresql.org/docs/current/errcodes-appendix.html) —
`57P01` and `57P03` cover the backend terminations that the logs show at
the same second as each burst.

Every occurrence in the logs lands in the same second as the source
connection dropping: a source restart refusing new connections, an
administrator terminating the backend, or an SSH tunnel closing.
Partitions for the same table succeed before and after each burst, which
rules out a genuine all-columns-excluded configuration. Confidence is
high: the pgx contract explains why the empty list appears, and the logs
confirm the trigger in each case. Nobody needs to act on the message
itself — the fix removes it, and the real connection error takes its
place. Two things worth a reviewer's attention: a permanently
misconfigured exclusion list would still produce the same message
legitimately, and that case belongs in validation rather than here; and
`SSH Tunnel Closed: EOF` is matched by the `io.EOF` branch before it
reaches the SSH-tunnel branch, which predates this change.

### Change

- `GetSelectedColumns` now checks `rows.Err()` after the iteration loop
and wraps it as `error getting selected columns for table %s`, matching
the check its sibling function in the same file already performs.
- All three callers — the `qrep` pull, table-schema fetch, and
validation — previously read a truncated or empty list as fact; they now
receive the underlying error.
- The connection failures that caused these bursts already classify on
their own: `57P03` and a closed tunnel as `NOTIFY_CONNECTIVITY`, `57P01`
as `NOTIFY_TERMINATE`, a bare `unexpected EOF` as `IGNORE_EOF`. No
classifier change is needed.

### Verification

- `TestGetSelectedColumnsReportsMidStreamConnectionLoss` breaks a live
connection after the query is sent and asserts that `GetSelectedColumns`
returns an error rather than an empty column list. It fails against the
unfixed code with a nil error and zero columns, reproducing the
production symptom exactly.
- The full classifier suite is green, as is the Postgres connector
package.
- An internal check rebuilt each observed production failure with the
post-fix wrap chain and confirmed all of them classify into an existing
class; none fall through to `OTHER`.

Resolves DBI-979
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants